Introduction to Machine Learning

1. Introduction

Welcome to Introduction to Machine Learning . This unit provides an overview of the main concepts and topics that we will study in this course. We begin by defining machine learning and understanding how it differs from traditional rule-based problem solving. We then look at the major types of machine learning problems, including supervised and unsupervised learning. The unit also introduces the distinction between classification and regression, the curse of dimensionality, and the main steps involved in a machine learning project. These concepts provide the foundation for the algorithms and techniques covered in the later units.

2. Theory

2.1 What is Machine Learning?

There are several definitions of machine learning, but two well-known definitions are given by Arthur Samuel and Tom Mitchell:

Together, these definitions highlight an important idea: a machine learning system improves its performance by learning from experience or data rather than relying only on rules explicitly written by a programmer.

2.2 How Humans vs. Machines Make Decisions

Humans make decisions in two primary ways:

• Logic & Reasoning: Carefully analyzing features and rules (e.g., comparing car specs within a budget)
• Experience: Learning from past observations (e.g., asking friends about their car experiences to decide)

Supervised machine learning is closer to this second approach. Instead of manually writing rules for every possible situation, we provide the system with previous examples and allow it to learn a model from those examples. We can think of this process in three simple stages: Remember — Formulate — Predict framework:

2.3 Three Major Types of Machine Learning

Machine learning problems are commonly divided into three major paradigms: supervised learning, unsupervised learning, and reinforcement learning. The main difference between them is how the learning system receives information and feedback.

2.4 Supervised Learning: Classification vs. Regression

Once we focus on supervised learning, another important distinction is needed. Supervised learning problems can be divided into classification and regression based on the type of output that the model needs to predict.

Side-by-Side Problem Examples

2.5 The Curse of Dimensionality

Machine learning datasets can contain a large number of features. When the number of features becomes very large, we face what is known as the curse of dimensionality . The high-dimensional data (hundreds or thousands of features) creates two critical problems:

1. Computational challenge: Processing time grows exponentially with dimension count
2. Performance degradation: Data becomes sparse in high-dimensional spaces, making it harder for models to generalize

One way to address these problems is dimensionality reduction . The basic idea is to keep the most useful information while reducing unnecessary or redundant features. This can also make the data easier to visualize, process, and analyze. Throughout this course, we will explore various dimensionality reduction techniques, including Principal Component Analysis (PCA) and feature selection methods.

2.6 Machine Learning Project Lifecycle

Learning individual machine learning algorithms is only one part of building a machine learning system. In practice, a machine learning project involves several steps, starting from collecting the raw data and ending with deploying and monitoring the final model. These steps are connected. Decisions made during data preparation affect model development, and mistakes made early in the process can lead to misleading evaluation results. For this reason, it is important to follow a proper workflow rather than treating model training as an isolated step. Every real-world ML project follows a structured 9-step lifecycle. Skipping or misordering steps is a common source of project failure.

Step 1: Raw Data Collection
Gathering data from various sources—databases, APIs, sensors, user interactions, etc. This raw data may come in different formats and may require integration from multiple sources.

Step 2: Initial Dataset Creation (Rows × Columns)
Organizing the collected data into a structured format where each row represents an example (instance) and each column represents a feature (attribute). At this stage, we have the basic dataset but haven't processed it further.

Step 3: Preprocessing Pipeline 1 – Data Cleaning (Performed on the Full Dataset)
Before we split the data, we fix universal data integrity issues. Because these operations correct formatting, structure, or obvious errors without using statistical properties of the target variable, they are safely applied to the entire dataset:

• Imputation (basic): Handling missing values—for example, removing completely empty rows or filling missing strings with "Unknown". (Note: sophisticated imputation using mean/median belongs in Step 5, as it requires the training distribution).
• Encoding: Converting categorical variables into numerical format (e.g., one‑hot encoding).
• Outlier detection: Identifying and handling extreme values that could skew the model (e.g., capping values at reasonable limits).
• Derived features: Creating new features from existing ones (e.g., combining age and income to create a "purchasing power" feature).

Step 4: Data Splitting (Train / Validation / Test)
Dividing the dataset into separate subsets. A common high-level split is 80% for training and 20% for testing. However, to support the model development loop in Step 6, we further split the training portion to carve out a validation set (or use cross‑validation techniques).

Step 5: Preprocessing Pipeline 2 – Advanced (Strictly Train-Only)
Any operation that learns statistical properties from the data must be performed here, after the split:

• Scaling/normalization (e.g., StandardScaler – learns mean and standard deviation)
• Dimensionality reduction (e.g., PCA – learns principal components)
• Sophisticated imputation (e.g., filling missing values with the training set's median)
• Feature selection (e.g., selecting features based on training-set variance)

Step 6: Model Development Loop
Using the validation set (derived from the training split in Step 4) to iteratively improve the model without ever peeking at the final test set:

• Hyperparameter search: Trying different model configurations
• Train: Training the model on the training set
• Validate: Evaluating performance on the validation set (or using cross‑validation within the training fold)
• Repeat: Iterating to improve performance based on validation feedback

Step 7: Final Model Training (Full Train Set)
After finding the best hyperparameters through the development loop, train the final model using the entire training set (including any validation data) with those optimal parameters.

Step 8: Test Set Evaluation
The test set, which has been completely untouched throughout the entire development process, is now used for the final evaluation. This provides the true estimate of how well the model will generalize to new, unseen data.

Step 9: Production Deployment & Monitoring
Deploying the model to a production environment where it can make predictions on real-world data, while continuously monitoring its performance and watching for drift or degradation over time.

The separation between the two preprocessing stages and the train/test split is particularly important. Basic data cleaning is performed before the split, while operations that learn parameters from the data, such as scaling and PCA, are handled after the split.

2.7 Course Roadmap at a Glance

The topics in this course are organized around the main types of machine learning problems and the techniques used to solve them. We begin with classification and related concepts, then move to regression, clustering, dimensionality reduction, and neural networks. The roadmap shows where these topics appear during the course and how they relate to the broader machine learning workflow.

3. Interactive Examples

Example 1: Classification vs. Regression Identifier

For each scenario below, classify whether it is a Classification or Regression problem. Click the button to reveal the answer.

Example 2: Project Lifecycle Ordering Quiz

Using the dropdowns, place the 9 lifecycle steps in the correct order. Then click Check Order to see how you did.

4. Numerical Solutions

Problem 1: Classifying Problem Types

For each of the following five tasks, determine (i) the ML paradigm (Supervised / Unsupervised / Reinforcement), and (ii) the sub-type (Classification / Regression / Clustering / N/A).

Problem 2: The Mitchell Framework (T, E, P)

A hospital is building a model to readmit or not readmit patients within 30 days of discharge. It uses 50,000 historical records with known readmission outcomes. The model is judged on its overall percentage of correct predictions on a held-out validation set.

Problem 3: Train / Test Splitting Check

A dataset has 2,500 labeled samples. We use an 80/20 train/test split.

5. Try It Yourself

6. Interactive Quiz

Answer all 5 MCQs. Click on an option to get instant feedback.